The comparison of normalization procedures based on different classification systems

In this paper, we develop a novel methodology within the IDCP measuring framework for comparing normalization procedures based on different classification systems of articles into scientific disciplines. Firstly, we discuss the properties of two rankings, based on a graphical and a numerical approach, for the comparison of any pair of normalization procedures using a single classification system for evaluation purposes. Secondly, when the normalization procedures are based on two different classification systems, we introduce two new rankings following the graphical and the numerical approaches. Each ranking is based on a double test that assesses the two normalization procedures in terms of the two classification systems on which they depend. Thirdly, we also compare the two normalization procedures using a third, independent classification system for evaluation purposes. In the empirical part of the paper we use: (i) a classification system consisting of 219 sub-fields identified with the Web of Science subject-categories; an aggregate classification system consisting of 19 broad fields, as well as a systematic and a random assignment of articles to sub-fields with the aim of maximizing or minimizing differences across sub-fields; (ii) four normalization procedures that use the field or sub-field mean citations of the above four classification systems as normalization factors, and (iii) a large dataset, indexed by Thomson Reuters, in which 4.4 million articles published in 1998-2003 with a five-year citation window are assigned to sub-fields using a fractional approach. The substantive results concerning the comparison of the four normalization procedures indicate that the methodology can be useful in practice. Acknowledgements. This is the second version of a Working Paper with the same title that appeared in this series in February 2013. The authors acknowledge financial support by Santander Universities Global Division of Banco Santander. Ruiz-Castillo also acknowledges financial help from the Spanish MEC through grant ECO2010-19596. Conversations with Ludo Waltman are greatly appreciated. However, all remaining shortcomings are the authors' sole responsibility.